1. Introduction and Goals

This project analyses the LendingClub's loan dataset. The dataset includes loan application data, the information on granted loans and their status.

The goal of this project is build several ML models which can be used to predict:

• Binary loan_status prediction model. We have used XGBoost and CatBoost to build a model which can be used to predict how likely an applicant is to default on a loan based on their information and the loan size, type and other specifics. This model can be used to decide whether the application should be approved or used by a third party investor to estimate risk. The goal of this model is to maximize the quality of probability prediction rather than the classification accuracy itself. In addition to estimating the default likelihood we'll also use this model to estimate:
  • Loan grades (A,B,C etc.) based on probability threshold, this does not directly attempt to predict the grades defined in the dataset but rather can be used as an alternative method to classify loan risk
  • Using this model we'll calculate the profitability and risk of a loan portfolio if only and provide an improved (more restrictive) method of evaluating loan applicants
• loan_grade prediction models, we'll evaluate and compare two different approaches
  • Multi classification model using XGBoost and CatBoost
  • Regression model that attempts to predict grade converted to a normalized numerical variable (predictions are rounded). While using a regression model for a classification problem is not ideal because it ignores the fact that spacing between different classes of ordinal features might differ (e.g. we don't know if the distance between grade A and B is the same as between B and C etc.)

Practical Applications

The default risk model can be used by potential investors to figure out which loans offered by the Lenders Club to invest in.

• e.g. a non experienced individual would likely base their decision by just looking into the grade, interest rate and maybe a few other variables and make their decision based on this.
• Our default risk model allows them to build an investment strategy based on their risk profile. Basically our goal is to provide a self-contained model that can be used (instead of just investing based on the grade/interest rate) to maximize returns based on acceptable risk.

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

1.1 EDA

This is the exploratory analysis of the dataset and includes all the features shown in the (Features) notebook.

The EDA was done in parallel with model design process to minimize the influence analyzing manually analyzing the interaction between features might have had on our model design process.

Data selection

We have three target variables which we'll try to predict using in our following sections. They are shown here as:

• target_loan_status: whether the loan applicant has defaulted on their loan (0 or 1)
• pass_grade: the grade assigned to the loan by the "Lenders Club" (A,B,C,D etc.)
• pass_interest_rate: the interest rate set for the loan (converted to float)

['target__loan_status']

1.1.1 Correlation Matrix

Loan Status has strong correlation with:

• Applicant FICO (credit) score
• Term of the loan (longer loans are somewhat more to be defaulted on)
• While the correlation with the zip_code is almost 0.4 it is not significant (other variables like grade, sub_grade and interest rate are not included in our model because they are calculated based on estimated loan risk, we could still include them if our goal is to verify the quality of "Lenders Club" estimates we chose not to include them to build a "clean" alternative model using the same source data).

Loan Grade and Interest Rate have significant correlations with a much larger number of variables:

• Loan Term
• Loan amount and installment size
• FICO score
• Loan purpose
• Applicant income
• Income verification status etc.

This would indicate that the Lenders Club is trying to use a large number of variables to determine the grade and interest rate but that those variables are not necessarily as useful at predicting the likelihood of default.

Because the datatypes of features vary we had to use different methods to measure the strength and significance of each pair:

- Chi-Squared Test: Assesses independence between two categorical variables.  For bool-bool pairs due to categorical nature.

- Point Biserial Correlation: Measures correlation between a binary and a continuous variable. For bool-numerical pairs to account for mixed data types.

- Spearman's Rank Correlation: Assesses monotonic relationship between two continuous variables. Used for numerical-numerical pairs (for non-normally distributed data).

Since the Chi-Squared test outputs an unbound statistic/value which can't be directly compared to  pointbiserialr or Spearman Rank we have converted them to a  ` Cramér's V:` value which is normalized between 0 and 1. This was done to make the values in the matrix more uniform however we must note that Cramér's V and Spearman's correlation coefficients are fundamentally different statistics and generally can't be directly compared.

1.1.1 Relationships between individual variables

This part includes various charts showing the relationships between specific manually selected variables:

We can see that much fewer grade A and B loans were defaulted on while at C or below the likelihood of default becomes higher than average:

'Default Rate and Expected Return by Grade:'

	default_rate	mean_interest_rate	expected_return
pass__grade
A	0.07	7.09	6.60
B	0.13	10.65	9.24
C	0.22	14.00	10.87
D	0.31	17.79	12.23
E	0.38	21.16	13.03
F	0.50	25.12	12.46
G	0.57	27.74	11.98

The expected_return show the rate of return adjusted by the default rate. e.g. while almost 60% of all G grade loans are defaulted on you can still expect to make ~12% per year because the interest is very high. A grade loans tend to be relatively much less profitable despite the much lower risk.

Text(0.5, 1.025, 'Grade and Interest Rate')

There is quite a bit of overlap between interest rates for lower grades (D+).

Text(0.5, 1.025, 'Grade and Loan Amount')

Text(0.5, 1.025, 'Grade and Annual Income')

As we would expect the applications by individuals who tend to have very low incomes tend to receive lower grades. However, the rate of E grade loans increases significantly along with grade A > ~100k yearly income which is quite interesting and requires additional analysis because this does not apply to loans of other grades (we can see in charts below that the income of ~25% applicants is > 90k so this can't really be explained by random variation due to small sample size)

Text(0.5, 1.025, 'Grade and Fico Score')

Text(0.5, 1.025, 'Default Rate and Interest Rate')

Text(0.5, 1.025, 'Default Rate and Annual Income')

While low income individuals are much more likely to default the risk of default doesen't seems to decrease much or at as income goes up.

Text(0.5, 1.025, 'Loan Amount and Purpose')

term_parser__term: int64
target__loan_status: float64
pass__loan_amnt: float64
pass__int_rate: float64
pass__grade: object
pass__home_ownership: object
pass__annual_inc: float64
pass__purpose: object
pass__fico_range_high: float64
pass__last_fico_range_high: float64

	term_parser__term
1233152	60
461729	36
738295	60
390716	36
350813	60
...	...
701647	36
1093222	36
1231982	36
143267	36
208225	36

5361 rows × 1 columns

term_parser__term_binned             category
term_parser__term                       int64
target__loan_status_binned           category
target__loan_status                   float64
pass__loan_amnt_binned               category
pass__loan_amnt                       float64
pass__int_rate_binned                category
pass__int_rate                        float64
pass__annual_inc_binned              category
pass__annual_inc                      float64
pass__fico_range_high_binned         category
pass__fico_range_high                 float64
pass__last_fico_range_high_binned    category
pass__last_fico_range_high            float64
pass__grade                            object
pass__home_ownership                   object
pass__purpose                          object
dtype: object

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

'term_parser__term_binned vs pass__fico_range_high No significant difference found (p-value = 0.302)'

None

None

None

None

None

None

None

None

1233152    714.0
461729     689.0
738295     694.0
390716     704.0
350813     694.0
           ...  
701647     719.0
1093222    749.0
1231982    719.0
143267     759.0
208225     664.0
Name: pass__fico_range_high, Length: 5361, dtype: float64

1.2.1 Variable Selection

1.2.2 Population Stability Index

We can use PSI to drop unstable features:

Interpreting PSI Values PSI = 0: No change in the distribution between the two datasets. 0 < PSI < 0.1: Indicates a minor change in distribution. Generally acceptable for most applications. 0.1 ≤ PSI < 0.25: Suggests a medium change, warranting investigation. PSI ≥ 0.25: Represents a significant change in the feature distribution, suggesting that the feature's behavior has altered considerably from the baseline. Features with high PSI are often considered unstable and may be dropped from the model to maintain its performance.

5361

DropHighPSIFeatures(split_frac=0.6)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

{'term_parser__term': 0.0,
 'emp_length_parser__emp_length': 0.00741761565898279,
 'target__loan_status': 0.0,
 'pass__loan_amnt': 0.025024201814442776,
 'pass__int_rate': 0.07176711417547209,
 'pass__installment': 0.011609246883005436,
 'pass__annual_inc': 0.03241740868044229,
 'pass__dti': 0.015406464288949512,
 'pass__delinq_2yrs': 0.0057098047008174195,
 'pass__fico_range_low': 0.024401601815981836,
 'pass__fico_range_high': 0.024401601815981836,
 'pass__inq_last_6mths': 0.0004806433251511103,
 'pass__open_acc': 0.020049125901470677,
 'pass__pub_rec': 0.0011239749466162331,
 'pass__revol_bal': 0.012375403776342789,
 'pass__revol_util': 0.010228320444652932,
 'pass__total_acc': 0.012150588779396194,
 'pass__last_fico_range_high': 0.018082420124089962,
 'pass__last_fico_range_low': 0.018082420124089962,
 'pass__collections_12_mths_ex_med': 0.0,
 'pass__inq_fi': 0.013248028575812698,
 'pass__inq_last_12m': 0.024401074697258687,
 'pass__chargeoff_within_12_mths': 0.0,
 'pass__mort_acc': 0.009757189838540613,
 'pass__pub_rec_bankruptcies': 4.868286594901147e-05,
 'pass__tax_liens': 0.0,
 'pass__tot_hi_cred_lim': 0.01626241771088676,
 'pass__total_bal_ex_mort': 0.020408531871268523}

2. Model Design

As we've discussed at the begining our goal is to build ML models to predict these target features:

• Loan Default Risk (loan_status)
• Loan Grade (loan_grade)
• Interest Rate (int_rate)

In all cases we've either used XGBoost or CatBoost models (based on performance and training time).

2.1. Model Pipelines

We've used sci-kit lear pipelines to build and design our models. The data passed to the pipeline is preprocessed separately in src/preproc which filters only completed loans (based on loan_status) and performs other clean up and preprocessing (without fundamentally transforming or scaling etc.) to make all the features compatible with our ML models.

Example of a pipeline:

Pipeline(steps=[('preprocessing',
                 FunctionTransformer(func=<function convert_to_category at 0x7f9a9fcefeb0>)),
                ('feat_trans_loan_grade', LoanGradeTransformer()),
                ('feat_trans_dti_inc_joint', JointApplicationTransformer()),
                ('feat_trans_loan_grade_risk', LoadGradeRiskTransformer()),
                ('feat_trans_fico_score', FICOScoreTransformer()),
                ('feat_trans_delinquency', Del...
                ('feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH',
                 DummyDropAllButFICOHigh()),
                ('feat_trans_dummytransformer',
                 DummyTestTransformer(option=<Options.OFF: 0>)),
                ('remove_columns',
                 FunctionTransformer(func=<function get_pipeline.<locals>.remove_columns_with_prefix at 0x7f9b2668d990>,
                                     kw_args={'prefix': 'target__'})),
                ('model',
                 <catboost.core.CatBoostClassifier object at 0x7f9a9bb53e80>)])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Each pipeline consists of these steps:

• a preprocessing function which handles data encoding if necessary (we do not encode categorical variables because XGBoost and CatBoost support them directly).
• a set of feature transformers selected based on hyper tuning results (the transformers were designed iteratively while designing the model and performing the EDA). The transformer merge/combine or otherwise modify various features based on subject knowledge (e.g. calculating various ratios or merging variables which represent basically the same features etc.)
• ML model.

2.2 Hyper-tuning

Because the dataset is very large we have split the tuning and model evaluation into separate notebooks, so they can be performed independently:

• Hyper tuning is first performed using default or fixed model hyperparameters to select the optimal transformers and their parameters.
• Hyperparameter tuning is performed using a fixed pipeline with transformers selected in the previous step.

This is a compromise approach, ideally we would use two nested loops with grid search performed on the outer level to select the transformers and a nested loop (using random or bayesian search) for model hyper tuning. However, considering the size of the dataset (and even if we use a relatively large sample) would be very slow because it would exponentially increase the number of required iterations.

The results are serialized to a binary file that includes the information on the model and training dataset.

2.3 Model Evaluation

Is performed by loading the hyper tuning information saved in the previous step, the model (with the same pipeline) is trained using the test training sample of the dataset, validated using CV and then evaluating using a separate training sample.

2.4 Production Deployment

The model from the previous step is serialized using (dill or joblib) and deployed to remote VM running a FastAPI backend that can be integrated into a website or other application.

2.1 Predicting Rejected/Accepted Loans

The model uses a dataset merged which combines accepted_2007_to_2018q4.csv and rejected_2007_to_2018Q4.csv (all feature the columns shared by both datasets are included).

Dropping transformer for ['term'], not in df

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

We have excluded all of the rows in the rejected applications dataset after 2013-11-05 because that applications after that data no longer used a standard FICO score which might make it hard to directly compare to older applications or accepted applications (which only contain a FICO score). The target values are in the target__Accepted columns (0=rejected, 1=accepted):

	Total Rows	Proportion
1	452597	0.62
0	279951	0.38

emp_length_parser__emp_length     float64
target__Accepted                    int64
zip__zip_code                    category
pass__loan_amnt                   float64
pass__addr_state                 category
pass__last_fico_range_high        float64
pass__dti                         float64
dtype: object

2.1. Tuning the Binary Default Risk Classifier

We'll tune multiple XGBoost model which predict the target__Accepted column using F1 and PR-AUC (we have tried other metrics however they generally provide the same or inferior results)

The best_score is the metric used for tuning by that specific pipeline (i.e. either F1 or PR-AUC so the values are not directly comparable):

	best_score	best_params
model_key
XGBoostTuneF1Rejected	0.934973	{'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 1, 'model__n_estimators': 200, 'model__min_child_weight': 1.5, 'model__max_depth': 4, 'model__learning_rate': 0.7, 'model__gamma': 0.1}
XGBoostTunePRAUCRejected	0.971341	{'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 5, 'model__n_estimators': 200, 'model__min_child_weight': 3, 'model__max_depth': 4, 'model__learning_rate': 0.2, 'model__gamma': 0.4}

Parameter hypertuning results:

'XGBoostTuneF1Rejected:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
10	2.004274	0.065828	0.145811	0.020581	2	200	3.0	4	0.5	0.05	0.945542	0.950051	0.945546	0.948917	0.950438	0.948099	0.002145	1
64	5.174102	0.263869	0.283433	0.063670	1	200	1.5	7	0.2	0.30	0.944586	0.951035	0.944873	0.949722	0.948728	0.947789	0.002605	2
35	2.533273	0.256556	0.187760	0.060749	1	100	3.0	7	0.2	0.40	0.945421	0.949731	0.943144	0.949221	0.948223	0.947148	0.002495	3
23	3.560320	0.184197	0.254836	0.034706	1	150	2.0	6	0.4	0.30	0.943171	0.948724	0.944632	0.950150	0.948988	0.947133	0.002721	4
4	2.197416	0.017642	0.146295	0.008489	1	250	2.0	4	0.6	0.10	0.944789	0.948810	0.943411	0.947546	0.948626	0.946636	0.002160	5
27	7.066529	0.537316	0.556671	0.125327	5	200	3.0	10	0.3	0.00	0.943887	0.946497	0.944256	0.948339	0.947579	0.946112	0.001770	6
28	1.686542	0.090513	0.121822	0.034204	1	100	1.5	6	0.3	0.30	0.943194	0.947666	0.943523	0.947964	0.947254	0.945920	0.002106	7
37	3.330606	0.207039	0.169529	0.057174	1	150	1.0	6	0.2	0.10	0.942733	0.946084	0.941687	0.946584	0.945464	0.944510	0.001940	8
45	1.970316	0.247999	0.133128	0.058171	1	150	1.0	4	0.5	0.30	0.941157	0.946166	0.941496	0.945741	0.946553	0.944223	0.002381	9
20	12.181362	0.853656	0.714233	0.144785	3	200	1.5	12	0.1	0.40	0.940956	0.945464	0.942960	0.944385	0.944187	0.943590	0.001539	10

'XGBoostTunePRAUCRejected:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
68	3.110086	0.069025	0.185129	0.027797	5	250	3.0	6	0.1	0.10	0.977639	0.976641	0.972051	0.976298	0.975465	0.975619	0.001915	1
5	2.122073	0.053020	0.131392	0.002833	2	250	2.5	4	0.4	0.10	0.976074	0.975842	0.973541	0.975383	0.973545	0.974877	0.001112	2
41	1.646022	0.025546	0.102047	0.002108	2	150	2.5	5	0.4	0.40	0.976477	0.974573	0.971847	0.975104	0.974569	0.974514	0.001505	3
9	1.420142	0.028590	0.085791	0.003710	10	100	3.0	6	0.4	0.30	0.975425	0.974433	0.970353	0.973632	0.973908	0.973550	0.001712	4
18	1.832829	0.025054	0.116294	0.004785	3	150	2.5	6	0.1	0.00	0.975046	0.974516	0.969263	0.973450	0.973666	0.973188	0.002046	5
1	6.367722	0.148925	0.344158	0.016222	3	150	2.0	12	0.1	0.05	0.974109	0.974901	0.970735	0.972796	0.973161	0.973140	0.001410	6
45	3.035600	0.051488	0.177038	0.009375	1	100	1.5	10	0.2	0.30	0.973911	0.973715	0.970719	0.973719	0.973546	0.973122	0.001207	7
48	0.836106	0.028394	0.049582	0.001505	1	50	2.0	7	0.3	0.30	0.974550	0.973497	0.968527	0.973081	0.973494	0.972630	0.002108	8
27	2.709189	0.064343	0.169259	0.005357	10	150	2.5	7	0.5	0.40	0.974966	0.971597	0.969468	0.972784	0.972234	0.972210	0.001779	9
12	1.168750	0.015082	0.071326	0.001318	5	100	3.0	5	0.7	0.00	0.973763	0.972154	0.968395	0.972478	0.973351	0.972028	0.001907	10

2.1.1 Training and Evaluating the Accepted/Rejected Application Model

The dataset is pre-split into a 20% test sample and a 80% training sample.

Dropping transformer for ['term'], not in df

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Total Samples Loaded: 732548

Training: XGBoostTuneF1 with: {'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 1, 'model__n_estimators': 200, 'model__min_child_weight': 1.5, 'model__max_depth': 4, 'model__learning_rate': 0.7, 'model__gamma': 0.1}
XGBoostTuneF1Rejected: 116.8 seconds
Training: XGBoostTunePRAUC with: {'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 5, 'model__n_estimators': 200, 'model__min_child_weight': 3, 'model__max_depth': 4, 'model__learning_rate': 0.2, 'model__gamma': 0.4}
XGBoostTunePRAUCRejected: 95.3 seconds

The models were trained using 5-Fold CV validation. We've evaluated them using a 20% test split:

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1009: UserWarning: The figure layout has changed to tight
  plt.tight_layout()

We have been able to achiever very solid results both when tuning for F1 and PR-AUC with F1=~0.9 and PR-AUC =~0.98.

We can see that our model is signficantly better at predicting positive (i.e. loan accepted) than negative cases, with quite a few rejected cases being misclassified as accepted. We could potentially use different hyper-parameters to optimize for precision instead of overal predictive power if our business problem required that (we'll see some example of that when building the Default Risk model in part 2.2

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

2.2 Predicting Default Risk

Our goal is to build a default risk estimation model that could be used by potential investors to build a strategy for selecting the optimal loans to invest in on the platform.

2.2.1 Tuning the Binary Default Risk Classifier

We'll tune multiple XGBoost model which predict the loan_status column using F1 and PR-AUC (we have tried other metrics but they gennerally provide the same or inferior results)

Dropping cols where: loan_status is missing 226070 -> 226067

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Tuning Results

The table below shows the highest target score (either F1 or PR-AUC) achieved for each model configuration:

	best_score	best_params
model_key
XGBoostF1Base	0.769397	{'model__scale_pos_weight': 3, 'model__n_estimators': 50, 'model__min_child_weight': 2, 'model__max_depth': 10, 'model__learning_rate': 0.05, 'model__gamma': 0.05}
XGBoostTuneF1	0.773237	{'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 2, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1DummyDropFICO	0.397436	{'feat_trans_dummy_drop_FICO__option': 1, 'model__scale_pos_weight': 5, 'model__n_estimators': 100, 'model__min_child_weight': 2, 'model__max_depth': 5, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostPRAUCBase	0.765507	{'model__scale_pos_weight': 10, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 12, 'model__learning_rate': 0.3, 'model__gamma': 0.3}
XGBoostTunePRAUC	0.829786	{'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05}

Preprocessing and Transformer Tuning

We have attempted to performance various feature transformations* to our dataset which could potentially improve performance (i.e. binning or merging certain variables etc.). Because we have a high number of different combinations and because invidual transformers might perform differently when tuning for certain metrics we've also included them in our tuning process included:

1. Grid search is performed with every combination of transformers using "semi-optimal" (based on previous tuning) hyperparameter defined for each specific model type.
2. Random search is used to select the optimal hyperparameter

• see src/feature_engineering/py for full list of transformers.

XGBoostF1Base and XGBoostPRAUCBaseuse a minimally processed (only clean up and some encoding without changing the structure) dataset.

XGBoostF1Base and XGBoostTuneF1 additionally use severeal feature transformers (that combine several features, calculate their ratios or use target encoding for specific categorical columns, see feature_engineering.py). Only the transformers that resulted in a signficant performance improvement are included.

XGBoostTuneF1DummyDropFICO is a baseline model which was trained after excluded the FICO score columns (which have the highest importance score for other models)

While we have tried LightGBM, CatBoost and other models initially (see shared/ml_config_core for various tested configurations) we've narrowed to these two configurations because they provided the best (or approximately the same as) results while also being the fastest to train.

The parameters tuned to maximize F1 generally provide the best performance for binary classification, however the PR-AUC configuration results in more accurate probabilities because (which might be preferable because our goal is to build a model to select the loans to invest in)

Results based on transformer combinations

0 = disabled, some transformers have multiple different parameters (e.g. 1,2..)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 for model_key, model_data in tuning_results.items():
      2     transformer_tun_all_civ_results = model_data.transformer_tun_all_civ_results
      3     if transformer_tun_all_civ_results is not None:

NameError: name 'tuning_results' is not defined

We have built a simple regression model to evaluate the impact of each transformer and the hyperparameter. This is just to give a rough estimate and to allow comparing individual parameters (the values themselves are not particularly meaningful)

'Transformer impact for XGBoostTuneF1'

	Transformer_Option	Impact
0	feat_trans_DropLowImportanceFeatures__option_1	0.0000
1	feat_trans_loan_purpose__option_1	-0.0014

'Transformer impact for XGBoostTuneF1'

	Transformer_Option	Impact
14	model__min_child_weight_2.5	0.0344
15	model__min_child_weight_3	0.0342
11	model__min_child_weight_1	0.0340
12	model__min_child_weight_1.5	0.0334
13	model__min_child_weight_2	0.0284
25	model__learning_rate_0.4	0.0204
22	model__learning_rate_0.1	0.0190
10	model__min_child_weight_0.75	0.0164
9	model__min_child_weight_0.5	0.0139
28	model__learning_rate_0.7	0.0137

'Transformer impact for XGBoostTunePRAUC'

	Transformer_Option	Impact
0	feat_trans_DropLowImportanceFeatures__option_1	0.0000
1	feat_trans_loan_purpose__option_1	-0.0006

'Transformer impact for XGBoostTunePRAUC'

	Transformer_Option	Impact
15	model__min_child_weight_3	0.0266
14	model__min_child_weight_2.5	0.0252
13	model__min_child_weight_2	0.0237
22	model__learning_rate_0.1	0.0189
12	model__min_child_weight_1.5	0.0180
11	model__min_child_weight_1	0.0176
23	model__learning_rate_0.2	0.0175
21	model__learning_rate_0.05	0.0151
4	model__n_estimators_100	0.0145
10	model__min_child_weight_0.75	0.0108

Parameter hypertuning results:

'XGBoostF1Base:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
3	2.539655	0.033946	0.029509	0.000972	5	50	2.50	12	0.05	0.30	0.782170	0.766917	0.781803	0.763786	0.766696	0.772274	0.008007	1
44	0.343730	0.011107	0.020059	0.000772	3	50	3.00	4	0.01	0.30	0.785016	0.773770	0.773289	0.759934	0.767932	0.771988	0.008202	2
66	3.856601	0.091281	0.057155	0.001304	3	200	2.00	7	0.01	0.00	0.777414	0.771641	0.770237	0.760291	0.770270	0.769971	0.005515	3
43	1.857459	0.005767	0.041977	0.004333	3	100	3.00	10	0.20	0.05	0.775111	0.776093	0.769784	0.759340	0.751838	0.766433	0.009415	4
4	2.330039	0.062270	0.039948	0.003183	2	250	2.00	12	0.20	0.10	0.783394	0.772851	0.771898	0.740873	0.756458	0.765094	0.014850	5
39	0.534821	0.018117	0.022498	0.000576	3	50	1.50	5	0.10	0.40	0.779496	0.755663	0.773508	0.758621	0.757119	0.764881	0.009720	6
6	3.377790	0.073363	0.039932	0.000557	2	150	0.75	12	0.10	0.40	0.771062	0.774908	0.776065	0.750896	0.743879	0.763362	0.013334	7
51	1.372830	0.036773	0.042140	0.001267	3	200	3.00	5	0.20	0.30	0.772887	0.779809	0.762923	0.752577	0.747529	0.763145	0.012068	8
20	3.694322	0.180248	0.063709	0.003112	3	200	1.00	7	0.05	0.40	0.770318	0.767748	0.775044	0.745234	0.752475	0.762164	0.011349	9
56	5.730208	0.081080	0.069219	0.002336	5	200	1.00	12	0.10	0.05	0.768812	0.774955	0.770209	0.743567	0.748593	0.761227	0.012635	10

'XGBoostTuneF1:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
7	0.963510	0.087027	0.033874	0.009908	2	50	2.5	7	0.10	0.40	0.791774	0.782684	0.779983	0.764706	0.759539	0.775737	0.011896	1
42	2.094696	0.097653	0.048334	0.006060	3	150	3.0	7	0.10	0.40	0.784722	0.770563	0.771105	0.760611	0.770740	0.771548	0.007681	2
53	1.193887	0.097053	0.041867	0.005184	3	200	2.5	4	0.10	0.00	0.778982	0.780811	0.766028	0.761364	0.764103	0.770258	0.008029	3
8	2.232088	0.126904	0.034808	0.001156	10	250	2.5	10	0.20	0.30	0.774138	0.774417	0.775758	0.759026	0.765018	0.769671	0.006550	4
32	1.556768	0.039411	0.033771	0.001642	3	250	2.5	12	0.40	0.05	0.783242	0.772686	0.774077	0.752389	0.751616	0.766802	0.012619	5
20	9.254182	0.111213	0.084302	0.002762	3	200	1.5	10	0.01	0.10	0.768028	0.766581	0.777391	0.756573	0.760181	0.765751	0.007164	6
17	1.689366	0.039251	0.043464	0.001895	5	150	3.0	10	0.40	0.00	0.774423	0.770723	0.777877	0.747598	0.756561	0.765436	0.011493	7
5	1.438166	0.083314	0.040884	0.004518	1	250	2.0	4	0.01	0.05	0.773709	0.779197	0.767377	0.764388	0.739336	0.764802	0.013725	8
51	4.991368	0.143159	0.075015	0.001778	1	250	2.5	10	0.05	0.05	0.778087	0.771084	0.765757	0.755677	0.751908	0.764503	0.009653	9
25	2.671416	0.080313	0.046523	0.002202	3	200	1.0	10	0.20	0.05	0.777174	0.776065	0.760436	0.756422	0.752328	0.764485	0.010240	10

'XGBoostTuneF1DummyDropFICO:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
60	0.482940	0.096194	0.025295	0.003457	3	50	2.5	4	0.10	0.30	0.378335	0.358429	0.367089	0.354430	0.348185	0.361293	0.010499	1
6	4.331436	0.090352	0.042518	0.001584	10	50	2.0	12	0.01	0.00	0.364597	0.355685	0.364871	0.346712	0.359761	0.358325	0.006724	2
50	4.146137	0.115205	0.095537	0.026142	3	250	2.0	6	0.01	0.10	0.377551	0.343830	0.354256	0.363636	0.348206	0.357496	0.012029	3
37	1.436562	0.053556	0.031436	0.001343	10	50	3.0	7	0.01	0.05	0.349868	0.349132	0.354515	0.356000	0.350133	0.351930	0.002777	4
33	16.386426	0.307811	0.166987	0.054258	10	200	0.5	12	0.01	0.00	0.343019	0.333096	0.316703	0.316948	0.346797	0.331313	0.012647	5
46	1.713501	0.065415	0.040049	0.002440	3	100	1.0	6	0.01	0.10	0.358156	0.310406	0.308113	0.320298	0.302412	0.319877	0.019993	6
13	0.523651	0.019858	0.024155	0.001896	3	50	2.0	5	0.40	0.30	0.323194	0.299065	0.302281	0.309160	0.306064	0.307953	0.008348	7
49	1.671080	0.074691	0.033222	0.001536	10	200	3.0	10	0.40	0.40	0.310511	0.301344	0.301850	0.289970	0.317121	0.304159	0.009197	8
12	0.556033	0.016523	0.023616	0.001066	10	50	1.5	5	0.90	0.00	0.312343	0.305506	0.284960	0.297297	0.303556	0.300732	0.009232	9
45	1.024397	0.105854	0.038746	0.002976	5	150	3.0	4	1.00	0.40	0.301035	0.300917	0.280702	0.309413	0.285171	0.295448	0.010763	10

'XGBoostPRAUCBase:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
41	4.712076	0.102368	0.083338	0.028096	3	250	3.0	7	0.01	0.10	0.788226	0.775678	0.771499	0.759392	0.771165	0.773192	0.009269	1
27	2.457221	0.134930	0.042669	0.002389	10	150	2.0	10	0.20	0.10	0.773451	0.762996	0.781222	0.761001	0.768260	0.769386	0.007335	2
8	1.363308	0.106730	0.038190	0.001965	3	150	2.5	5	0.05	0.05	0.781327	0.771987	0.769861	0.763578	0.759664	0.769283	0.007453	3
23	3.857984	0.076118	0.052538	0.008277	3	250	1.5	12	0.10	0.30	0.774194	0.780919	0.772242	0.754127	0.758109	0.767918	0.010134	4
42	2.214649	0.101345	0.052262	0.008496	10	200	2.0	7	0.20	0.10	0.777103	0.772021	0.773124	0.760034	0.755120	0.767480	0.008406	5
65	1.996158	0.045136	0.044282	0.001806	3	150	1.5	6	0.05	0.10	0.774194	0.767031	0.771574	0.765289	0.757417	0.767101	0.005786	6
62	5.784741	0.083577	0.077049	0.013330	3	200	1.0	10	0.05	0.40	0.771993	0.771041	0.778571	0.757840	0.751148	0.766119	0.010064	7
61	1.500142	0.040802	0.036259	0.001377	3	200	3.0	7	0.20	0.30	0.775439	0.776408	0.763766	0.750647	0.763276	0.765907	0.009439	8
40	1.545100	0.082604	0.028552	0.000674	3	250	2.0	12	0.40	0.10	0.786738	0.765077	0.756757	0.749350	0.771611	0.765906	0.012846	9
47	0.764223	0.023834	0.025007	0.000622	2	50	2.5	7	0.20	0.40	0.783760	0.772208	0.767442	0.738488	0.754579	0.763295	0.015536	10

'XGBoostTunePRAUC:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
46	1.167071	0.107811	0.034444	0.015000	5	50	3.0	7	0.10	0.00	0.843684	0.835045	0.835531	0.809094	0.820630	0.828797	0.012336	1
45	5.331813	0.371688	0.085230	0.024038	1	200	2.5	10	0.05	0.05	0.835988	0.838556	0.830562	0.805956	0.823805	0.826973	0.011662	2
35	7.442352	0.189858	0.072972	0.003792	2	200	3.0	12	0.01	0.30	0.847388	0.829016	0.832578	0.797764	0.819548	0.825259	0.016407	3
1	4.444903	0.116943	0.057768	0.002373	5	150	2.5	10	0.05	0.40	0.832729	0.834509	0.831418	0.801665	0.822907	0.824646	0.012162	4
39	0.572882	0.066626	0.027122	0.007576	5	50	3.0	5	0.20	0.00	0.837891	0.834781	0.822234	0.808294	0.815544	0.823749	0.011227	5
27	2.394500	0.106129	0.053726	0.004209	2	250	2.0	7	0.10	0.40	0.828421	0.832822	0.827069	0.805578	0.818790	0.822536	0.009619	6
38	0.901095	0.013271	0.031447	0.001048	3	150	2.0	4	0.10	0.10	0.831417	0.832216	0.822399	0.802439	0.814818	0.820658	0.011121	7
55	2.220437	0.304225	0.034987	0.004498	10	250	3.0	12	0.20	0.40	0.828787	0.832201	0.820685	0.801097	0.815254	0.819605	0.011006	8
51	6.164659	0.160574	0.085509	0.020234	2	200	1.0	10	0.05	0.40	0.832165	0.825742	0.823084	0.797151	0.818171	0.819262	0.011943	9
24	7.032639	0.165788	0.067873	0.002149	3	150	1.0	12	0.05	0.00	0.838111	0.823868	0.823744	0.798606	0.806830	0.818232	0.013946	10

2.2.2 Training and Testing the Default Risk Model

The dataset is pre-split into a 20% test sample and a 80% training sample (10% of which is used for tuning):

Dropping cols where: loan_status is missing 2034631 -> 2034601

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Training: XGBoostTuneF1 with: {'model__scale_pos_weight': 5, 'model__n_estimators': 50, 'model__min_child_weight': 2.5, 'model__max_depth': 12, 'model__learning_rate': 0.05, 'model__gamma': 0.3}
XGBoostF1Base: 133.8 seconds
Training: XGBoostTuneF1 with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 0, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.1, 'model__gamma': 0.4}
XGBoostTuneF1: 64.1 seconds
Training: XGBoostTuneF1 with: {'feat_trans_dummy_drop_FICO__option': 1, 'model__scale_pos_weight': 3, 'model__n_estimators': 50, 'model__min_child_weight': 2.5, 'model__max_depth': 4, 'model__learning_rate': 0.1, 'model__gamma': 0.3}
XGBoostTuneF1DummyDropFICO: 42.2 seconds
Training: XGBoostTuneF1 with: {'model__scale_pos_weight': 3, 'model__n_estimators': 250, 'model__min_child_weight': 3, 'model__max_depth': 7, 'model__learning_rate': 0.01, 'model__gamma': 0.1}
XGBoostPRAUCBase: 217.9 seconds
Training: XGBoostTunePRAUC with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 0, 'model__scale_pos_weight': 5, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 7, 'model__learning_rate': 0.1, 'model__gamma': 0}
XGBoostTunePRAUC: 62.9 seconds

2.2.3 Results and Evaluation

Predicting default risk (loan_status) seems to be significantly harder than rejected/accepted loans. However, we were still able to achieve relatively good results (F1=~0.85).

The figure layout has changed to tight

Training: XGBoostTunePRAUC with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05} XGBoostTunePRAUC: 1.1 seconds

The classification performance of all the models using the default 50% threshold is comparable (XGBoostTuneF1DummyDropFICO was included as a baseline).

We have chosen the XGBoostTunePRAUC configuration as our "production model" since it performs the best at predicting accurate probabilities (i.e. lowest log_loss).

XGBoostPRAUCBase (which achieves similar classification performance) uses a minimally processed dataset while XGBoostTunePRAUC uses several feature transformers (that were selected based on their impact on performances as part of our tuning process):

• DropLowImportanceFeatures, dropping features with importance close to 0.0 simplifies the model and has no negative impact on performance
• InstallementIncomeRatio, calculates the ratio between the income of the application and the expected installment size
• LoanPurposeRiskTransformer, uses target encoding for the Loan Purpose column by calculating the default risk based on it.

We have tried multiple other feature transformers (see src/feature_engineering.py) but they had negative or no signficant positive impact on performance.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

Evaluating the Best Performing (XGBoostTunePRAUC) Model

Average Default Rate by Grade in the Dataset:

	Counts	Proportion	Loan_Status_Ratio
A	211381	0.175	0.060
B	353495	0.292	0.134
C	343327	0.284	0.224
D	180840	0.149	0.304
E	84224	0.070	0.384
F	28719	0.024	0.451
G	8218	0.007	0.502

Evaluating Financial Returns/Performance

We'll use the model we have built to narrow down the sample of loans that have a lower risk to default.

This would be more useful to potential investors rather than the Lenders Club itself, they can use it in addition to the grade and other data provided by the Lenders Club to select the loans which not to invest it in.

The table below shows the total returns of all the loans in the dataset. We have to note that this is highly simplified because it assumes that all the loans that were defaulted on had to be completely written off but realistically this is not really the case:

• all lenders are likely to make at least some payments before they default
• defaulted loans can still be sold to third part for a fraction of the owed amount In future version of our model we should take these factors into account.

	Total Loan Amount	Total Interest Paid	Expected Return %	Total Return %	Total Loss
0	17451.88m	1770.28m	13.645651	10.143752	3757.82m

We have selected only the loans which are unlikely to default using the default 50% threshold (note that the predicted probabilities don't directly indicate the likelihood of default):

	Actual	Hypothetical
Total Loan Amount	17451.8813M	2455.6286M
Total Interest Paid	1770.2756M	310.9426M
Total Return %	10.14%	12.66%
Default Rate	19.96%	2.29%
Total Loss	3757.8181M	57.4278M
Losses Avoided	None	3700.3903M
Interest Lost	None	-1459.3330M
Total Applications Accepted	1210204	172025

Our model is fairly conservative and has rejected most of the loans offered by the Lenders Club (i.e. more than 80% are considered to be too risky). This would make it not very useful for the Lenders Club itself (however we might be able to improve this tuning the threshold). However, from the perspective of potential investors this is perfectly acceptable because they are likely to only invest into a small subset of available loans.

Using our model would allow investors to signficantly improve their returns by 3% or so, further improvements might be possible by improving the threshold (we'll look into this in the next section).

Calculating Probability Thresholds

If we're using our binary classification model to assign grades to loans based on default probabilities we can't directly try to predict the grades from the source dataset.

Basically what we can do is to use the source dataset as a baseline and try to determine whether our model can outperform the method the Lending Club is using for determine grades, we'll try to maintain the same ratio of grades while trying to minimize the default risk for loans included in top grades (basically we want to make sure that's the spread of default risk between grades is maximized i.e. default risk for grade A is as close to 0% and for grade F as close to 100% as possible while also ensuring that the risk increased with continuously with grade)

Attempt to set non-positive xlim on a log-scaled axis will be ignored.

The chart shows the performance of the if only individual with stroke Prob. > T are selected. Additionally the overlay indicates the number of people whose predicted P is in an given range. The overlays can be used to selected the most at risk individual based on the probability predicted for them

We have used the predicted probabilities to suggest alternative grades for each loan. Expected D.R is the default rate based on original grades. We can see our grades are much better at excluding higher risk loans from higher grades.

Generally this does not necessarily mean that source grades are inaccurate or objectively worse, the investors club is likely using a more straightforward and transparent algorithm for assigning them and investors should be using additional criteria when selecting which loans to invest anyway.

Comparing Interest Rates for Predicted Grades

In this section we'll try to evaluate investment strategies that are entirely based on loan grades (either the original ones or ones provided by our model)

Predicted Grade Real Return Rates

'Returns and Default Rate by Grade in Source Dataset:'

	default_rate	mean_interest_rate	expected_return
pass__grade_merged
A	0.06	7.11	6.68
B	0.14	10.69	9.23
C	0.22	14.03	10.87
D	0.30	17.73	12.37
E-F-G	0.41	22.47	13.35

'Returns and Default Rate Based on Default Probability Model:'

	default_rate	mean_interest_rate	expected_return
predicted_grades
A	0.003	10.519	10.485
B	0.009	12.168	12.063
C	0.083	13.951	12.796
D	0.558	14.732	6.507
E-F-G	0.902	16.846	1.652

We can see that while using our model to invested in the worst grades loans would lead to relatively poor returns investing in only in C grade or higher rated loans would provide better returns than using the Lenders Club's values to invest in the same loans:

• Investing only in C loans would provide almost the same returns as the far riskier E-F-G loans using the original grades.
• Only investing in A loans still yields ~10%, considering that the default rate is extremely low it might still be a preferable option to investing into high risk loans.

Basically using our model the investor would be able to minimize default risk while still getting decent returns i.e. if they only invested in loans with an estimated 10% default likelyhood or less using the original grades (so only A grade loans) they'd only make 6-7% while using our model this can be increased to more than 10%.

Limitations

We must take into account that this calculation is very simplified and therefore probably not accurate because, our model and calculations assume that Loss Given Default (LGD) is 100% and that Exposure at Default (EAD) is also 100% which is not accurate: - loans usually are not defaulted on immediately (i.e. some payments are almost always received) - most applicants who are unable to keep up with their payments are given some leeway and {PAYMENT PLAN} - failed loans are often sold to third parties for at least some fraction of the owed amount

Therefore, to provide a more accurate estimate and to figure out whether our model would actually be useful we'd need to incorporate the actual amounts received etc.

Total samples loaded : 1000

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

2    288
1    283
0    210
3    137
4     82
Name: target__grade, dtype: int64

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostOrdinalGradeRegressor_Default n_iters=50 with:
 {'model__learning_rate': [0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12, None], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [-0.88042996 -0.87518446         nan         nan         nan         nan
 -0.84446263 -0.85909352         nan         nan         nan         nan
 -0.88042996 -0.87518446         nan         nan         nan         nan
 -0.84446263 -0.85909352         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -0.89201032 -0.85979639         nan         nan         nan         nan
 -0.86618961 -0.84521511         nan         nan         nan         nan
 -0.89201032 -0.85979639         nan         nan         nan         nan
 -0.86618961 -0.84521511         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -0.88678709 -0.88353414         nan         nan         nan         nan
 -0.8461568  -0.85376689         nan         nan         nan         nan
 -0.88678709 -0.88353414         nan         nan         nan         nan
 -0.8461568  -0.85376689         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -0.90117584 -0.86268431         nan         nan         nan         nan
 -0.86789756 -0.84720845         nan         nan         nan         nan
 -0.90117584 -0.86268431         nan         nan         nan         nan
 -0.86789756 -0.84720845         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 50 candidates, totalling 250 fits
Best Model Parameters: {'model__n_estimators': 150, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostOrdinalGradeRegressor_Default Fit, total time:36.8

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostMulticlassTunePRAUC n_iters=50 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [0.39865792 0.38915238        nan        nan        nan        nan
 0.39865792 0.38915238        nan        nan        nan        nan
 0.39865792 0.38915238        nan        nan        nan        nan
 0.39865792 0.38915238        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
 0.3892805  0.37535732        nan        nan        nan        nan
 0.3901784  0.38983167        nan        nan        nan        nan
 0.3892805  0.37535732        nan        nan        nan        nan
 0.3901784  0.38983167        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
 0.40028704 0.38580882        nan        nan        nan        nan
 0.40028689 0.38949293        nan        nan        nan        nan
 0.40028704 0.38580882        nan        nan        nan        nan
 0.40028689 0.38949293        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 50 candidates, totalling 250 fits
Best Model Parameters: {'model__n_estimators': 150, 'model__min_child_weight': 0.5, 'model__max_depth': 12, 'model__learning_rate': 0.2, 'model__gamma': 0}
XGBoostMulticlassTunePRAUC Fit, total time:101.2

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostMulticlassTuneLogLoss n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [-1.33395429 -1.33670064         nan         nan         nan         nan
 -1.33395429 -1.33670064         nan         nan         nan         nan
 -1.33395429 -1.33670064         nan         nan         nan         nan
 -1.33395429 -1.33670064         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -1.33603874 -1.3392099          nan         nan         nan         nan
 -1.33488397 -1.34099195         nan         nan         nan         nan
 -1.33603874 -1.3392099          nan         nan         nan         nan
 -1.33488397 -1.34099195         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -1.33462439 -1.33826324         nan         nan         nan         nan
 -1.33244639 -1.33767518         nan         nan         nan         nan
 -1.33462439 -1.33826324         nan         nan         nan         nan
 -1.33244639 -1.33767518         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__n_estimators': 100, 'model__min_child_weight': 2.5, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0}
XGBoostMulticlassTuneLogLoss Fit, total time:118.8

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostF1Multiclass n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [0.442 0.441   nan   nan   nan   nan 0.442 0.441   nan   nan   nan   nan
 0.442 0.441   nan   nan   nan   nan 0.442 0.441   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
 0.431 0.421   nan   nan   nan   nan 0.427 0.429   nan   nan   nan   nan
 0.431 0.421   nan   nan   nan   nan 0.427 0.429   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
 0.437 0.436   nan   nan   nan   nan 0.437 0.436   nan   nan   nan   nan
 0.437 0.436   nan   nan   nan   nan 0.437 0.436   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
 0.435 0.422   nan   nan   nan   nan 0.442 0.425   nan   nan   nan   nan
 0.435 0.422   nan   nan   nan   nan 0.442 0.425   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}
XGBoostF1Multiclass Fit, total time:302.4

	best_score	best_params
model_key
XGBoostOrdinalGradeRegressor_Default	-1.304155	{'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostMulticlassTunePRAUC	0.407976	{'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 0.5, 'model__max_depth': 12, 'model__learning_rate': 0.2, 'model__gamma': 0}
XGBoostMulticlassTuneLogLoss	-1.287763	{'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 100, 'model__min_child_weight': 2.5, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0}
XGBoostF1Multiclass	0.439000	{'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}

'\n Model: XGBoostOrdinalGradeRegressor_Default'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	-1.248444
169	1	1	1	0	0	0	1	-1.248444
168	1	1	1	0	0	0	0	-1.248444
127	1	0	1	0	1	0	1	-1.248444
126	1	0	1	0	1	0	0	-1.248444
121	1	0	1	0	0	0	1	-1.248444
120	1	0	1	0	0	0	0	-1.248444
174	1	1	1	0	1	0	0	-1.248444
78	0	1	1	0	1	0	0	-1.248444
73	0	1	1	0	0	0	1	-1.248444

'Transformer impact for XGBoostOrdinalGradeRegressor_Default'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0176
5	feat_trans_new_dti_after_loan__option_1	0.0082
3	feat_trans_fico_score__option_1	0.0000
1	feat_trans_dti_inc_joint__option_1	-0.0011
0	feat_trans_delinquency__option_1	-0.0022
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.3815

XGBoostOrdinalGradeRegressor_Default

'\n Model: XGBoostMulticlassTunePRAUC'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	0.244104
169	1	1	1	0	0	0	1	0.244104
168	1	1	1	0	0	0	0	0.244104
127	1	0	1	0	1	0	1	0.244104
126	1	0	1	0	1	0	0	0.244104
121	1	0	1	0	0	0	1	0.244104
120	1	0	1	0	0	0	0	0.244104
174	1	1	1	0	1	0	0	0.244104
78	0	1	1	0	1	0	0	0.244104
73	0	1	1	0	0	0	1	0.244104

'Transformer impact for XGBoostMulticlassTunePRAUC'

	Transformer_Option	Impact
0	feat_trans_delinquency__option_1	0.0019
4	feat_trans_inst_income_ratio__option_1	0.0016
3	feat_trans_fico_score__option_1	0.0000
1	feat_trans_dti_inc_joint__option_1	-0.0019
5	feat_trans_new_dti_after_loan__option_1	-0.0049
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1474

XGBoostMulticlassTunePRAUC

'\n Model: XGBoostMulticlassTuneLogLoss'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	-1.485756
169	1	1	1	0	0	0	1	-1.485756
168	1	1	1	0	0	0	0	-1.485756
127	1	0	1	0	1	0	1	-1.485756
126	1	0	1	0	1	0	0	-1.485756
121	1	0	1	0	0	0	1	-1.485756
120	1	0	1	0	0	0	0	-1.485756
174	1	1	1	0	1	0	0	-1.485756
78	0	1	1	0	1	0	0	-1.485756
73	0	1	1	0	0	0	1	-1.485756

'Transformer impact for XGBoostMulticlassTuneLogLoss'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0002
1	feat_trans_dti_inc_joint__option_1	0.0000
3	feat_trans_fico_score__option_1	0.0000
0	feat_trans_delinquency__option_1	-0.0003
5	feat_trans_new_dti_after_loan__option_1	-0.0033
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1490

XGBoostMulticlassTuneLogLoss

'\n Model: XGBoostF1Multiclass'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	0.33
169	1	1	1	0	0	0	1	0.33
168	1	1	1	0	0	0	0	0.33
127	1	0	1	0	1	0	1	0.33
126	1	0	1	0	1	0	0	0.33
121	1	0	1	0	0	0	1	0.33
120	1	0	1	0	0	0	0	0.33
174	1	1	1	0	1	0	0	0.33
78	0	1	1	0	1	0	0	0.33
73	0	1	1	0	0	0	1	0.33

'Transformer impact for XGBoostF1Multiclass'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0012
3	feat_trans_fico_score__option_1	0.0000
0	feat_trans_delinquency__option_1	-0.0003
5	feat_trans_new_dti_after_loan__option_1	-0.0035
1	feat_trans_dti_inc_joint__option_1	-0.0067
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1040

XGBoostF1Multiclass

Dropping cols where: grade is missing 90000 -> 89996

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Total samples loaded : 89996

Pipeline(steps=[('preprocessing',
                 FunctionTransformer(func=<function convert_to_category at 0x7f1aa69abd00>)),
                ('feat_trans_loan_grade', LoanGradeTransformer()),
                ('feat_trans_dti_inc_joint', JointApplicationTransformer()),
                ('feat_trans_fico_score', FICOScoreTransformer()),
                ('feat_trans_delinquency', DelinquencyTransformer()),
                ('feat_trans_inst_income_ratio', Ins...
                               feature_types=None, gamma=0.1, grow_policy=None,
                               importance_type=None,
                               interaction_constraints=None, learning_rate=0.3,
                               max_bin=None, max_cat_threshold=None,
                               max_cat_to_onehot=None, max_delta_step=None,
                               max_depth=7, max_leaves=None,
                               min_child_weight=2.5, missing=nan,
                               monotone_constraints=None, multi_strategy=None,
                               n_estimators=150, n_jobs=None,
                               num_parallel_tree=None,
                               objective='multi:softprob', ...))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

<Figure size 1100x900 with 0 Axes>

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1471: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.200921
1	emp_length_parser__emp_length	0.000000
2	zip__zip_code	0.014028
3	pass__loan_amnt	0.019779
4	pass__installment	0.028261
5	pass__home_ownership	0.000000
6	pass__annual_inc	0.010391
7	pass__verification_status	0.052370
8	pass__purpose	0.033422
9	pass__addr_state	0.000000
10	pass__dti	0.016340
11	pass__delinq_2yrs	0.000000
12	pass__fico_range_low	0.203827
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.041348
15	pass__mths_since_last_delinq	0.000000
16	pass__mths_since_last_record	0.000000
17	pass__open_acc	0.000000
18	pass__pub_rec	0.000000
19	pass__revol_bal	0.020648
20	pass__revol_util	0.038275
21	pass__total_acc	0.028199
22	pass__initial_list_status	0.046777
23	pass__last_fico_range_high	0.089089
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.000000
26	pass__mths_since_last_major_derog	0.000000
27	pass__application_type	0.000000
28	pass__verification_status_joint	0.000000
29	pass__inq_fi	0.015884
30	pass__inq_last_12m	0.017836
31	pass__chargeoff_within_12_mths	0.000000
32	pass__mort_acc	0.016369
33	pass__pub_rec_bankruptcies	0.000000
34	pass__tax_liens	0.000000
35	pass__tot_hi_cred_lim	0.019794
36	pass__total_bal_ex_mort	0.000970
37	installment_income_ratio	0.053909
38	new_dti	0.031560

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.271462
1	emp_length_parser__emp_length	0.008289
2	zip__zip_code	0.106345
3	pass__loan_amnt	0.061711
4	pass__installment	0.046212
5	pass__home_ownership	0.008468
6	pass__annual_inc	0.012985
7	pass__verification_status	0.016868
8	pass__purpose	0.023965
9	pass__addr_state	0.015981
10	pass__dti	0.015480
11	pass__delinq_2yrs	0.011471
12	pass__fico_range_low	0.085221
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.023751
15	pass__mths_since_last_delinq	0.009172
16	pass__mths_since_last_record	0.010692
17	pass__open_acc	0.008931
18	pass__pub_rec	0.009866
19	pass__revol_bal	0.010371
20	pass__revol_util	0.010980
21	pass__total_acc	0.010002
22	pass__initial_list_status	0.048485
23	pass__last_fico_range_high	0.020814
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.013943
26	pass__mths_since_last_major_derog	0.009580
27	pass__application_type	0.013948
28	pass__verification_status_joint	0.014838
29	pass__inq_fi	0.014440
30	pass__inq_last_12m	0.014292
31	pass__chargeoff_within_12_mths	0.013176
32	pass__mort_acc	0.010901
33	pass__pub_rec_bankruptcies	0.012358
34	pass__tax_liens	0.013203
35	pass__tot_hi_cred_lim	0.012728
36	pass__total_bal_ex_mort	0.009067

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.186552
1	emp_length_parser__emp_length	0.008174
2	zip__zip_code	0.066410
3	pass__loan_amnt	0.069225
4	pass__installment	0.054875
5	pass__home_ownership	0.008708
6	pass__annual_inc	0.017367
7	pass__verification_status	0.024303
8	pass__purpose	0.027819
9	pass__addr_state	0.014011
10	pass__dti	0.019013
11	pass__delinq_2yrs	0.010187
12	pass__fico_range_low	0.124549
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.032111
15	pass__mths_since_last_delinq	0.009185
16	pass__mths_since_last_record	0.010386
17	pass__open_acc	0.008975
18	pass__pub_rec	0.007683
19	pass__revol_bal	0.011625
20	pass__revol_util	0.013283
21	pass__total_acc	0.011336
22	pass__initial_list_status	0.055525
23	pass__last_fico_range_high	0.029334
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.011566
26	pass__mths_since_last_major_derog	0.010041
27	pass__application_type	0.018512
28	pass__annual_inc_joint	0.011335
29	pass__dti_joint	0.017017
30	pass__verification_status_joint	0.013634
31	pass__inq_fi	0.017480
32	pass__inq_last_12m	0.016551
33	pass__chargeoff_within_12_mths	0.006341
34	pass__mort_acc	0.012275
35	pass__pub_rec_bankruptcies	0.012554
36	pass__tax_liens	0.008045
37	pass__tot_hi_cred_lim	0.014740
38	pass__total_bal_ex_mort	0.009274

While the performance for A,B,C grades is relatively acceptable (F1 > ~0.8) performance when predicting the lower grades is very poor (especially for grade G which is almost never classified correctly).

As we have learnt when building our default risk model there difference in returns and other features does not vary as much for lower quality grades which might make it hard to distinguish them. Therefore we'll use the same approach and merge E-F-G grades into a single group.

Merging E-F-G loan grades into a single group

Training: XGBoostF1Multiclass with: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}
XGBoostF1Multiclass: 60.8 seconds

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.200921
1	emp_length_parser__emp_length	0.000000
2	zip__zip_code	0.014028
3	pass__loan_amnt	0.019779
4	pass__installment	0.028261
5	pass__home_ownership	0.000000
6	pass__annual_inc	0.010391
7	pass__verification_status	0.052370
8	pass__purpose	0.033422
9	pass__addr_state	0.000000
10	pass__dti	0.016340
11	pass__delinq_2yrs	0.000000
12	pass__fico_range_low	0.203827
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.041348
15	pass__mths_since_last_delinq	0.000000
16	pass__mths_since_last_record	0.000000
17	pass__open_acc	0.000000
18	pass__pub_rec	0.000000
19	pass__revol_bal	0.020648
20	pass__revol_util	0.038275
21	pass__total_acc	0.028199
22	pass__initial_list_status	0.046777
23	pass__last_fico_range_high	0.089089
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.000000
26	pass__mths_since_last_major_derog	0.000000
27	pass__application_type	0.000000
28	pass__verification_status_joint	0.000000
29	pass__inq_fi	0.015884
30	pass__inq_last_12m	0.017836
31	pass__chargeoff_within_12_mths	0.000000
32	pass__mort_acc	0.016369
33	pass__pub_rec_bankruptcies	0.000000
34	pass__tax_liens	0.000000
35	pass__tot_hi_cred_lim	0.019794
36	pass__total_bal_ex_mort	0.000970
37	installment_income_ratio	0.053909
38	new_dti	0.031560

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.271462
1	emp_length_parser__emp_length	0.008289
2	zip__zip_code	0.106345
3	pass__loan_amnt	0.061711
4	pass__installment	0.046212
5	pass__home_ownership	0.008468
6	pass__annual_inc	0.012985
7	pass__verification_status	0.016868
8	pass__purpose	0.023965
9	pass__addr_state	0.015981
10	pass__dti	0.015480
11	pass__delinq_2yrs	0.011471
12	pass__fico_range_low	0.085221
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.023751
15	pass__mths_since_last_delinq	0.009172
16	pass__mths_since_last_record	0.010692
17	pass__open_acc	0.008931
18	pass__pub_rec	0.009866
19	pass__revol_bal	0.010371
20	pass__revol_util	0.010980
21	pass__total_acc	0.010002
22	pass__initial_list_status	0.048485
23	pass__last_fico_range_high	0.020814
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.013943
26	pass__mths_since_last_major_derog	0.009580
27	pass__application_type	0.013948
28	pass__verification_status_joint	0.014838
29	pass__inq_fi	0.014440
30	pass__inq_last_12m	0.014292
31	pass__chargeoff_within_12_mths	0.013176
32	pass__mort_acc	0.010901
33	pass__pub_rec_bankruptcies	0.012358
34	pass__tax_liens	0.013203
35	pass__tot_hi_cred_lim	0.012728
36	pass__total_bal_ex_mort	0.009067

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.186552
1	emp_length_parser__emp_length	0.008174
2	zip__zip_code	0.066410
3	pass__loan_amnt	0.069225
4	pass__installment	0.054875
5	pass__home_ownership	0.008708
6	pass__annual_inc	0.017367
7	pass__verification_status	0.024303
8	pass__purpose	0.027819
9	pass__addr_state	0.014011
10	pass__dti	0.019013
11	pass__delinq_2yrs	0.010187
12	pass__fico_range_low	0.124549
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.032111
15	pass__mths_since_last_delinq	0.009185
16	pass__mths_since_last_record	0.010386
17	pass__open_acc	0.008975
18	pass__pub_rec	0.007683
19	pass__revol_bal	0.011625
20	pass__revol_util	0.013283
21	pass__total_acc	0.011336
22	pass__initial_list_status	0.055525
23	pass__last_fico_range_high	0.029334
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.011566
26	pass__mths_since_last_major_derog	0.010041
27	pass__application_type	0.018512
28	pass__annual_inc_joint	0.011335
29	pass__dti_joint	0.017017
30	pass__verification_status_joint	0.013634
31	pass__inq_fi	0.017480
32	pass__inq_last_12m	0.016551
33	pass__chargeoff_within_12_mths	0.006341
34	pass__mort_acc	0.012275
35	pass__pub_rec_bankruptcies	0.012554
36	pass__tax_liens	0.008045
37	pass__tot_hi_cred_lim	0.014740
38	pass__total_bal_ex_mort	0.009274

Classifying Subgrades

Dropping cols where: sub_grade is missing 90000 -> 89996

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Total samples loaded : 89996

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2922: UserWarning: The y_pred values do not sum to one. Starting from 1.5 thiswill result in an error.
  warnings.warn(

Training: XGBoostF1Multiclass with: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}
XGBoostF1Multiclass: 293.7 seconds

Predicting Sub-grades

We've attempted to build a model which predicts individual sub-grades (e.g. A1, A2 ... G5) in addition to top level grades. An XGBoost multi-classificaiton model was used, however the performance was unsatisfactory. We've included the overal performance summary and feature importances below. However, we decide to not provide an indepth analysis because the model would not be useful for any practical applications (a different approach would probably suit this problem better due to the high number of classes and their ordinal nature)

XGBoostF1Multiclass
{'f1': 0.199, 'accuracy': 0.2733, 'precision': 0.2108, 'recall': 0.1947, 'log_loss': 2}
None

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.106485
1	emp_length_parser__emp_length	0.016260
2	zip__zip_code	0.063670
3	pass__loan_amnt	0.048561
4	pass__installment	0.046285
5	pass__home_ownership	0.017024
6	pass__annual_inc	0.020558
7	pass__verification_status	0.024662
8	pass__purpose	0.026665
9	pass__addr_state	0.025046
10	pass__dti	0.021775
11	pass__delinq_2yrs	0.017929
12	pass__fico_range_low	0.065871
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.027946
15	pass__mths_since_last_delinq	0.017716
16	pass__mths_since_last_record	0.019174
17	pass__open_acc	0.016289
18	pass__pub_rec	0.020212
19	pass__revol_bal	0.018316
20	pass__revol_util	0.019093
21	pass__total_acc	0.017927
22	pass__initial_list_status	0.033912
23	pass__last_fico_range_high	0.024567
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.016248
26	pass__mths_since_last_major_derog	0.017968
27	pass__application_type	0.026973
28	pass__annual_inc_joint	0.021111
29	pass__dti_joint	0.022676
30	pass__verification_status_joint	0.022423
31	pass__inq_fi	0.023245
32	pass__inq_last_12m	0.022318
33	pass__chargeoff_within_12_mths	0.018417
34	pass__mort_acc	0.017800
35	pass__pub_rec_bankruptcies	0.020849
36	pass__tax_liens	0.016260
37	pass__tot_hi_cred_lim	0.020096
38	pass__total_bal_ex_mort	0.017674

Conclusions

Limitations and Future Improvements

• We've applied relatively few feature engineering techniques in our project. While the rejected loans dataset is fairly simple trying to do that when using a complex model like XGBoost might not be necessary however the features in the accepted loans dataset are significantly more complex.

  • In addition to our feature transformers and using model importancess/SHAP plots to eliminate features we should use additional techniques (we've used PSI to eliminate unstable feature however that did not improve the performance so this might not be necessary with the dataset we have)

  • Clustering, might help us to identify inherent groupings or segments in the data which the model might be missing on its own. Clusters could can be used as new features, potentially capturing non-linear relationships that are not explicitly modeled in the original feature set.

  • Featuretools and other automated feature engineering packages.

• Tuning process can be improved by using bayesian tuning to speed it up (we've used Optuna initially)

• Model sizes, some of the models are very large, we should look into:

  • simplifying the XGBoost models themselves (by including less features and tweaking other parameters as long as that does not decrease performance)
  • using more efficient structures for serialization, we should consider optimizing the dataclasses used for storing the model and related metadata and consider using a prebuilt solution or a library instead of building our own custom methods.

Outcomes

Overal the performance of the rejected/accepted and loan default risk classifier model is acceptable however the multi classification model for predicting grades under-perform to a significant degree (however there unlikely to be any practical application for such models)

Appendices

Individual feature distributions and other supplementary data

4.1 Rejected/Accepted Loans Dataset

This is a supplement to the main EDA notebook (1.1 EDA) that includes the individual analysis of all numerical and categorical variables. The insights obtained here together with the analysis of interaction between individual variables was used to build the preprocessing pipeline and individual transformers (for feature engineering) that were used when building our models (see the EDA notebook for additional details).

Dropping transformer for ['term'], not in df

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

	emp_length_parser__emp_length	target__Accepted	zip__zip_code	pass__loan_amnt	pass__addr_state	pass__last_fico_range_high	pass__dti
133128	NaN	1	863	10000.0	AZ	824.0	0.2640
734065	2.0	1	208	6500.0	MD	819.0	0.0730
714506	11.0	1	275	8400.0	NC	709.0	0.1729
1866411	11.0	1	750	31000.0	TX	714.0	0.2701
3391910	0.0	0	769	7500.0	TX	656.0	0.4015

'Features with NaN Values:'

	Total NaN Values	Proportion NaN (%)
emp_length_parser__emp_length	457	5.0
pass__last_fico_range_high	90	1.0
pass__dti	7	0.0

'Total Duplicate Rows: 0.06'

	emp_length_parser__emp_length	target__Accepted	zip__zip_code	pass__loan_amnt	pass__addr_state	pass__last_fico_range_high	pass__dti
2654687	0.0	0	286	5000.0	NC	0.0	0.00
2907914	0.0	0	100	35000.0	NY	NaN	-0.01
3087120	0.0	0	331	1000.0	FL	0.0	0.00
3043169	0.0	0	100	35000.0	NY	NaN	-0.01
2724606	0.0	0	331	1000.0	FL	0.0	0.00
3135218	0.0	0	286	5000.0	NC	0.0	0.00

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1046: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  axes[0].set_xticklabels(tick_labels)

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1036: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  count_plot_df[variable] = count_plot_df[variable].astype("str")

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1036: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  count_plot_df[variable] = count_plot_df[variable].astype("str")

4.2 Accepted Loans Dataset

This is a supplement to the main EDA notebook (1.1 EDA) that includes the individual analysis of all numerical and categorical variables. The insights obtained here together with the analysis of interaction between individual variables was used to build the preprocessing pipeline and individual transformers (for feature engineering) that were used when building our models (see the EDA notebook for additional details).

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

	term_parser__term	emp_length_parser__emp_length	target__grade	pass__loan_amnt	pass__int_rate	pass__installment	pass__sub_grade	pass__home_ownership	pass__annual_inc	pass__verification_status	pass__purpose	pass__zip_code	pass__addr_state	pass__dti	pass__delinq_2yrs	pass__fico_range_low	pass__fico_range_high	pass__inq_last_6mths	pass__mths_since_last_delinq	pass__mths_since_last_record	pass__open_acc	pass__pub_rec	pass__revol_bal	pass__revol_util	pass__total_acc	pass__initial_list_status	pass__last_fico_range_high	pass__last_fico_range_low	pass__collections_12_mths_ex_med	pass__mths_since_last_major_derog	pass__application_type	pass__annual_inc_joint	pass__dti_joint	pass__verification_status_joint	pass__inq_fi	pass__inq_last_12m	pass__chargeoff_within_12_mths	pass__mort_acc	pass__pub_rec_bankruptcies	pass__tax_liens	pass__tot_hi_cred_lim	pass__total_bal_ex_mort	pass__loan_status
2086299	36	1.0	0	14000.0	6.08	426.42	A2	MORTGAGE	70000.0	Not Verified	debt_consolidation	176xx	PA	31.10	0.0	800.0	804.0	0.0	NaN	NaN	18.0	0.0	22237.0	32.3	33.0	w	824.0	820.0	0.0	NaN	Joint App	350000.0	8.61	Not Verified	1.0	0.0	0.0	1.0	0.0	0.0	492126.0	78093.0	Current
1233152	60	NaN	4	35000.0	19.52	917.97	E2	MORTGAGE	120000.0	Verified	debt_consolidation	799xx	TX	23.75	1.0	710.0	714.0	3.0	22.0	NaN	11.0	0.0	49315.0	43.7	34.0	f	804.0	800.0	0.0	NaN	Individual	NaN	NaN	None	NaN	NaN	0.0	3.0	0.0	0.0	390414.0	90585.0	Fully Paid
461729	36	10.0	0	10000.0	7.34	310.33	A4	MORTGAGE	70000.0	Not Verified	debt_consolidation	063xx	CT	14.72	0.0	685.0	689.0	0.0	NaN	106.0	6.0	1.0	21943.0	47.5	16.0	w	749.0	745.0	0.0	NaN	Individual	NaN	NaN	None	0.0	0.0	0.0	1.0	1.0	0.0	56200.0	26022.0	Fully Paid
738295	60	10.0	3	21600.0	17.27	539.96	D2	MORTGAGE	96000.0	Verified	debt_consolidation	482xx	MI	15.83	1.0	690.0	694.0	1.0	15.0	NaN	11.0	0.0	14751.0	71.3	32.0	w	639.0	635.0	0.0	42.0	Individual	NaN	NaN	None	3.0	1.0	0.0	3.0	0.0	0.0	242183.0	76547.0	Fully Paid
864585	36	3.0	1	3300.0	10.07	106.60	B2	RENT	22000.0	Verified	other	490xx	MI	2.73	0.0	730.0	734.0	0.0	NaN	NaN	2.0	0.0	680.0	42.5	3.0	w	764.0	760.0	0.0	NaN	Individual	NaN	NaN	None	0.0	0.0	0.0	0.0	0.0	0.0	1600.0	680.0	Current

'Features with NaN Values:'

	Total NaN Values	Proportion NaN (%)
emp_length_parser__emp_length	1325	15.0
pass__dti	2	0.0
pass__mths_since_last_delinq	4587	51.0
pass__mths_since_last_record	7593	84.0
pass__revol_util	4	0.0
pass__mths_since_last_major_derog	6671	74.0
pass__annual_inc_joint	8545	95.0
pass__dti_joint	8545	95.0
pass__verification_status_joint	8556	95.0
pass__inq_fi	3422	38.0
pass__inq_last_12m	3422	38.0
pass__mort_acc	179	2.0
pass__pub_rec_bankruptcies	7	0.0
pass__tot_hi_cred_lim	275	3.0
pass__total_bal_ex_mort	179	2.0

	Total NaN Values	Proportion NaN (%)
pass__verification_status_joint	8556	95.0

count              444
unique               3
top       Not Verified
freq               212
Name: pass__verification_status_joint, dtype: object

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.